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1.0  SUMMARY 


Resolution  is  often  provided  as  one  of  the  key  parameters  addressing  the  quality  characteristic  of 
a  sensor.  One  traditional  approach  when  detennining  the  resolution  of  a  sensor/display  system  is 
to  use  a  resolution  target  pattern  to  detect  the  smallest  element  that  can  be  “resolved”  using  the 
system.  This  requires  a  human  in  the  loop  to  make  the  assessment.  This  study  investigated  the 
use  of  a  custom  designed  software  approach  to  generate  an  effective  resolution  value  for  a 
sensor.  Landolt  Cs  were  selected  as  the  resolution  target,  which  were  imaged  at  multiple 
distances  with  different  sensors.  The  images  were  analyzed  using  custom  software  to  determine 
the  orientation  of  the  C  at  each  distance,  which  resulted  in  a  probability  of  correct  orientation 
detection  curve  as  a  function  of  distance.  This  curve  was  used  to  generate  a  “resolution”  for  the 
sensor  without  involving  human  vision.  Resolution  results  for  four  different  spectral  band 
sensors  were  obtained  as  well  as  effective  resolution  of  fused  images  from  select  pairs  of  sensors. 
These  results  and  the  possible  use  of  this  synthetic  observer  resolution  approach  are  presented 
and  discussed,  as  well  as  possible  future  research  relating  this  resolution  to  human  visual 
perfonnance  with  fused  image  sources. 


2.0  INTRODUCTION 
2.1  Resolution  measurement  background  and  philosophy 

Over  the  years,  there  have  been  many  methods  developed  to  detennine  the  image  quality  of  an 
image-generating  system  such  as  a  sensor/display  combination1.  In  most  cases,  the  final 
consumer  of  the  image  produced  is  a  human  observer  using  their  visual  capability  to  extract 
visual  infonnation  from  the  displayed  image.  In  recent  years,  imaging  systems  and  image 
manipulation  have  moved  from  the  analog  world  to  the  digital  world,  which  has  probably  added 
a  bit  more  confusion  to  the  issue  of  image  quality  or  resolution.  In  general,  resolution  is  the 
ability  of  a  sensor/display  system  to  produce  detail;  the  higher  the  resolution  the  finer  the  detail 
that  can  be  displayed.  With  the  advent  of  digital  imagery  and  sensor  detectors  that  are  composed 
of  an  array  of  discrete  elements,  it  is  tempting,  and  not  entirely  wrong,  to  characterize  the 
resolution  of  the  system  by  the  number  of  picture  elements  (pixels)  for  the  display  or  sensor 
elements  in  the  case  of  the  sensor.  For  example,  VGA  resolution  for  computer  displays  is  480 
elements  high  by  640  elements  wide  and  SVGA  is  600x800  elements.  This  describes  the 
number  of  samples  that  can  be  displayed  but  it  says  nothing  of  the  quality  of  the  actual  display 
medium  characteristics  (luminance,  contrast  capability,  noise,  color,  refresh  rate,  active  area  to 
total  area  ratio,  etc.)  or  of  the  signal/infonnation  used  to  feed  the  individual  pixels.  Nevertheless, 
this  numerical  value  of  pixel  or  sensor  element  count  is  often  given  as  a  primary  metric  for  the 
resolution  (quality)  of  the  sensor  or  display. 

Another  common  approach  to  detennining  the  resolution  of  a  sensor/display  system  is  to  image 
an  appropriate  resolution  test  target  and  determine  the  smallest  sized  critical  test  pattern  element 
that  can  be  seen  by  a  human  observer2,3,4,5.  Many  test  patterns  have  been  developed  over  the 
years  such  as  gratings,  tri-bars,  tumbling  Es,  the  Snellen  chart,  and  the  Landolt  C  chart  to  test 
vision  or  to  test  imaging  systems  using  vision6.  The  test  pattern  typically  has  test  elements  of 
various  sizes  so  that  the  human  observer  can  pick  out  the  smallest  size  that  they  can  resolve.  In 
this  case,  one  needs  to  make  sure  that  the  resulting  test  image  size  on  the  display  is  well  within 
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human  vision  capability  so  that  what  is  really  being  tested  is  the  imaging  system  and  not  the 
visual  system  of  the  observer.  An  alternative  to  the  multi-sized  test  pattern  is  to  use  a  single  size 
test  element  but  image  it  at  various  distances  until  a  distance  is  obtained  at  which  the  test  object 
is  barely  resolved. 

For  this  effort,  we  selected  the  Landolt  C  as  the  resolution  target  and  we  used  distance  from  the 
sensor  to  the  target  as  the  method  for  probing  the  resolution  of  the  sensor.  We  also  selected  four 
sensors  representing  four  different  spectral  bands  for  resolution  measurement.  The  following 
section  provides  the  details  of  the  approach  to  making  semi-automated  resolution  measurements 
of  sensors  and  fused  images  produced  from  pairs  of  sensors. 


3.0  APPROACH 

As  noted  above,  our  basic  approach  was  to  obtain  images  of  a  Landolt  C  resolution  target  at 
different  distances  and  feed  these  images  into  a  custom  software  algorithm  to  see  how  well  the 
software  could  correctly  determine  the  orientation  of  the  C  as  a  function  of  distance.  From  these 
data  we  could  then  calculate  a  “resolution”  from  the  sigmoid  curve  fit  to  these  data.  Using  this 
approach  it  is  possible  to  determine  the  resolution  for  multispectral  fused  images  (a  synthetic 
sensor)  and  later,  compare  resolutions  determined  by  the  software  to  resolutions  determined  from 
human  testing.  The  following  sections  describe  this  approach  in  more  detail. 

3.1  Sensors  and  zoom  lane 

Table  1  is  a  summary  of  the  four  sensors  that  were  selected  for  this  study  along  with  their  basic 
characteristics.  The  four  spectral  bands  selected  were  long-wave  infrared  (LWIR),  short-wave 
infrared  (SWIR),  near  infrared  (NIR)  and  visible  (VIS).  The  basic  number  of  sensor  elements  is 
listed  in  column  4  of  Table  1  along  with  the  measured  horizontal  field  of  view  (column  5)  and 
the  calculated  individual  sensor  element  “resolution,”  which  was  determined  by  dividing  the 
horizontal  field  of  view  by  the  number  of  horizontal  sensor  elements  (and  then  converting  the 
results  to  milliradians).  The  calculated  sensor  “resolution”  (last  column  in  Table  1)  is  a 
reasonable  estimate  of  the  level  of  detail  that  the  sensor  should  be  able  to  detect.  It  is  this 
column  of  “resolution”  numbers  that  will  be  compared  with  the  sensor  resolutions  obtained  by 
the  Landolt  C  test  procedure  used  in  this  study.  Figure  1  shows  the  sensor  suite7  with  the  sensors 
mounted  as  close  together  as  physically  possible  to  minimize  registration  problems  with  the 
fused  images.  Figure  2  shows  the  sensor  suite  mounted  to  a  computer-controlled,  movable  cart 
mounted  on  a  track  that  can  be  positioned  at  selected  distances  from  the  resolution  target  pattern 
at  the  end  of  the  zoom  lane. 


Table  1.  The  four  sensors  used  in  this  study  and  their  relevant  characteristics 


Manufacturer 

Sensor 

Spectral  Band 

Array  Size 
(HxV) 

Measured  Horizontal 
FOV  (deg) 

Calculated 
Resolution  for 
Horizontal  (mrad) 

BAE 

LTC550 

LWIR  (7- 15  pm) 

320x240 

35.9 

1.96 

Sensors  Unlimited 

SU320 

SWIR  (0.9-2. 5  pm) 

320x240 

30.7 

1.67 

ITT 

ISA-780 

NIR  (0.665-0.9  pm) 

768x493 

31.2 

0.71 

Sony 

XC-73 

VIS  (0.4-0.75  pm) 

768x494 

33.2 

0.75 
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Figure  1.  Multispectral  suite  of  four  sensors 


Figure  2.  Sensor  suite  mounted  on  a  movable  cart 


3.2  Resolution  target  pattern  and  the  sensor  image  collection  procedure 

The  test  pattern  was  a  62.5mm  high/wide  Landolt  C  (gap  size  of  12.5mm)  that  was  mounted  to  a 
device  designed  to  provide  reasonable  spectral  energy  across  all  four  spectral  bands  of  the 
sensors  selected.  This  device,  dubbed  the  quad-emissive  display  (QED),  is  more  fully  described 
in  Pinkus  and  Task  (2009)7.  Figures  3  and  4  show  the  QED  with  the  Landolt  C  mounted  in 
place.  The  front  surface  is  illuminated  with  incandescent  light  appropriate  to  the  sensor(s)  being 
used  and  the  rear  plate  is  heated  to  provide  an  accurately  controlled  temperature  difference 
between  the  plastic,  white  front  surface  and  the  flat  black,  metallic  back  surface.  The  Landolt  C 
is  actually  a  stencil,  or  cut-out,  so  that  the  black  back  surface  shows  through  the  stencil  opening. 
The  Landolt  C  can  be  mounted  in  any  of  the  four  cardinal  positions  (gap  pointing  up,  right, 
down,  or  left). 


Figure  3.  QED  with  Landolt  C  (side) 


Figure  4.  QED  with  Landolt  C  (front) 


The  basic  procedure  was  to  position  the  movable  cart,  with  the  four  sensors  mounted  to  it,  at  a 
specific  distance  and  then  capture  images  from  all  four  sensors  at  the  same  distance.  This  was 
done  with  the  Landolt  C  oriented  in  each  of  the  four  cardinal  directions.  However,  preliminary 
testing  resulted  in  two  modifications  to  this  basic  procedure. 

First,  the  set  of  distances  used  for  the  “higher  resolution”  sensors  (the  VIS  at  0.75  mrad  and  the 
NIR  at  0.71  mrad)  were  different  (farther  away)  than  those  used  for  the  “lower  resolution” 
sensors  (the  LWIR  at  1.96  mrad  and  the  SWIR  at  1.67  mrad).  Table  2  is  a  summary  of  the 
distances  used  to  collect  images  for  each  of  the  sensors  along  with  the  calculated  resolution 
based  on  the  field  of  view  and  sensor  horizontal  element  count.  This  was  done  to  minimize  the 
number  of  images  that  needed  to  be  recorded  for  each  sensor.  It  was  expected  that  the  measured 
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resolution  of  each  sensor,  discussed  more  fully  below,  would  be  in  the  general  range  of  the 
previously  discussed  calculated  resolution.  Based  on  the  fundamental  concept  of  resolution  for 
the  Landolt  C,  one  would  expect  the  distance  at  which  the  gap  in  the  Landolt  C  could  be 
“resolved”  is  the  distance  at  which  the  angular  subtense  of  the  gap  is  approximately  equal  to  the 
calculated  resolution  of  the  sensor.  The  first  row  of  Table  2  lists  the  calculated  resolutions  for 
each  of  the  sensors,  which  can  be  compared  with  the  corresponding  angular  subtense  of  the  gap 
size  for  each  image  collection  distance.  Selecting  different  collection  distances  for  the  different 
sensors  reduced  the  overall  number  of  images  required  but,  it  also  made  it  impossible  to  look  at 
certain  fused  image  combinations,  such  as  VIS  with  LWIR,  since  there  were  insufficient  images 
taken  at  the  same  distance  with  the  same  sensor. 

Second,  it  was  determined  that  there  was  a  sampling  issue  with  respect  to  the  relative  position  of 
the  Landolt  C  gap  and  the  individual  sensor  elements.  Depending  on  where  the  Landolt  C  gap 
fell  with  respect  to  the  sensor  element,  the  C  might  be  more  or  less  interpretable.  This 
phenomenon  would  presumably  have  similar  effects  on  either  a  human  observer  or  a  software 
algorithm  intended  to  mimic  an  observer’s  visual  perfonnance  with  these  images.  In  order  to  at 
least  partially  offset  the  effect  just  described,  it  was  decided  to  capture  images  of  the  Landolt  C 
at  multiple  lateral  and  vertical  “shifts”  so  there  would  be  a  rather  large  collection  of  relative  C- 
gap/sensor  element  positions.  This  was  implemented  by  moving  the  QED  target  pattern  l/5th  of 
the  gap  size  (2.5mm)  in  both  vertical  and  horizontal  directions  resulting  in  a  5X5  matrix  of 
images  of  the  Landolt  C  for  each  orientation  at  each  distance.  Therefore,  there  were  100  images 
captured  for  each  sensor  at  each  distance  (five  lateral  shifts  times  five  vertical  shifts  times  four 
orientations). 


Table  2.  Summary  of  distances  used  for  each  sensor 


Sensor-calculated  Resolution 

Sensor-calculated  Resolution 

1 .96  mrad 

1.67  mrad 

0.71  mrad 

0.75  mrad 

LWIR 

SWIR 

C-gap  Size 

NIR 

VIS 

C-gap  Size 

(meters) 

(meters) 

(mrad) 

(meters) 

(meters) 

(mrad) 

6 

6 

2.08 

10 

10 

1.25 

7.5 

7.5 

1.67 

13 

13 

0.96 

9 

9 

1.39 

16 

16 

0.78 

11 

11 

1.14 

20 

20 

0.63 

13 

13 

0.96 

24 

24 

0.52 

16 

16 

0.78 

28 

28 

0.45 

19 

0.66 

34 

34 

0.37 

Each  of  the  four  sensors  provided  an  analog  signal  output  (RS-170)  that  was  fed  into  a  frame 
grabber  to  capture  a  single  frame  of  the  output.  The  frame  grabber  converted  the  signal  into  a 
single  frame  with  640  horizontal  by  480  vertical  elements.  Since  two  of  the  sensors  (VIS  and 
NIR)  had  somewhat  greater  sensor  element  counts  (768x494)  than  this  matrix,  it  is  probable  that 
the  frame  grabber  limited  the  measurement  of  resolution  for  these  sensors.  While  this  is  not  a 
desirable  situation,  it  does  not  detract  from  the  main  objective  of  this  project,  which  is  to  devise  a 
software  algorithm  that  will  produce  similar  resolution  assessment  results  as  a  panel  of  human 
observers.  Since  the  images  “viewed”  by  both  the  software  algorithm  and  the  human  observers 
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are  the  same,  the  fact  that  the  VIS  and  NIR  sensor  resolutions  might  be  measured  as  somewhat 
poorer  than  what  they  really  are  (due  to  the  frame-grabber  limitation)  does  not  take  away  from 
the  human  observer/software  algorithm  comparison.  However,  in  order  to  seriously  use  this 
technique  to  determine  sensor  resolution,  one  needs  a  frame  grabber  with  an  element  count 
comfortably  in  excess  of  any  sensor  being  tested.  All  images  captured  by  the  frame  grabber  were 
saved  in  BMP  format  for  analysis  by  the  software  algorithm. 

3.3  Sensor  resolution  determination 

The  software  developed  to  determine  the  C’s  orientation  has  been  named  ALCOR  which  stands 
for  Auto-Landolt  C  Orientation  Recognition.  The  ALCOR  software  needed  to  do  two  basic 
things:  1)  find  the  Landolt  C  in  the  image  and  2)  determine  the  orientation  of  the  Landolt  C.  Of 
these  two,  determining  the  orientation  of  the  Landolt  C  was  deemed  to  be,  by  far,  the  most 
important  since  it  is  this  function  that  is  used  to  determine  the  resolution  of  the  sensor.  However, 
this  function  could  not  be  achieved  until  after  the  Landolt  C  position  was  identified  in  the 
software.  To  facilitate  the  location  of  the  Landolt  C,  the  QED  target  pattern  generator  was 
positioned  so  that  it  would  be  approximately  in  the  center  of  each  of  the  images  captured  from 
the  sensors.  The  QED  target  pattern  generator  had  a  fairly  uniform  surround  around  the  Landolt 
C  stencil  (see  Figures  3  and  4)  to  aid  in  picking  out  the  Landolt  C  from  its  immediate  surround. 
The  central  100x100  pixels  were  searched  to  find  the  center  of  the  Landolt  C.  Once  the  center  of 
the  Landolt  C  was  identified,  the  collection  of  pixels  in  the  up,  down,  left,  and  right  directions 
were  analyzed  to  determine  the  direction  that  had  a  high  likelihood  of  containing  the  Landolt  C 
gap.  Since  the  Landolt  C  target  was  always  a  dark  (low  signal  level)  C  on  a  light  (high  signal 
level)  background,  the  gap  should  be  in  the  direction  of  the  highest  signal  level  (the  direction  that 
does  not  cross  over  the  lower  signal  level  C).  This  is  somewhat  analogous  to  a  radar  display 
sweep  checking  the  four  cardinal  directions  for  which  direction  has  the  highest  “return.” 

Once  the  software  determined  the  probable  direction  of  the  Landolt  C,  the  software  would 
compare  this  direction  to  the  correct  direction  (which  was  coded  into  the  image  file  name  along 
with  the  sensor  type  and  the  distance  from  which  it  was  recorded).  The  software  then  provided 
two  outputs.  One  was  a  summary  report  of  all  of  the  images  in  the  file  showing  the  number  of 
correct  determinations  of  orientation  for  each  distance  and  the  other  was  a  graphic  of  the  central 
set  of  pixels  of  each  image.  This  graphic  had  the  direction  vector  marked  in  green  if  the 
software-determined  direction  was  correct  or  in  red  if  the  software  was  wrong  (with  a  blue  vector 
indicating  the  correct  direction).  Samples  of  these  outputs  are  shown  in  Figures  5  and  6  for  a 
relatively  close  distance  (all  responses  were  correct)  and  a  farther  distance  (one  of  the  responses 
was  wrong)  respectively. 


0—  — o  0  L  tl 

Figure  5.  Graphic  output  from  the  ALCOR  software  showing  that  it  determined  the  Landolt  C  direction  correctly 
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BLUE 


Figure  6.  Example  of  graphic  output  of  the  ALCOR  software  in  which  it  got  the  first  orientation  wrong 
(it  guessed  the  red-dashed  line  direction  to  the  left  but  the  correct  orientation  was  the  blue  line  direction  to  the  right) 

Although  the  ALCOR  software  is  capable  of  looking  for  the  Landolt  C  orientation  in  eight 
directions  (up,  down,  left,  right  and  upper-left,  upper-right,  lower-left,  lower-right),  it  was 
limited  to  just  the  four  orientations  for  this  study. 

After  all  of  the  images  at  the  different  distances  were  assessed  by  the  ALCOR  software,  the 
percent  correct  was  graphed  against  the  distance  for  each  sensor.  To  determine  a  resolution 

o 

value  from  these  data,  it  was  necessary  to  fit  a  curve  to  the  data.  Logistic  regression  was 
selected  as  the  method  of  curve-fitting.  The  fonn  of  the  generalized  logistic  equation  used  for 
this  effort  is  shown  below. 


(V  -  k\  (l) 

Logiti  =  In  J  =  B1*Xi  +  B0 

where: 

p  =  probability  of  occurrence  (in  our  case,  the  probability  of  correct  C  orientation 
determination) 

k  —  constant  (in  this  case,  0.25,  which  is  the  probability  of  guessing  the  correct 
orientation  of  the  C) 

=  slope  determined  by  the  linear  regression  of  the  logit  with  distance 
B0  =  intercept  determined  by  the  linear  regression  of  the  logit  with  distance 
Xi  =  distance  from  sensor  to  Landolt  C  target 
Logiti  ~  the  percent  correct  at  distance  Xi  (for  this  case). 

The  values  of  B0  and  B{  were  determined  for  each  sensor  by  perfonning  a  linear  regression  (least 
squares  fit)  of  the  percent  correct  versus  distance.  Since  we  are  primarily  interested  in  the 
probability  of  correct  determination  of  the  Landolt  C  orientation  as  a  function  of  distance,  Equation 
1  needs  to  be  solved  for  p.  Equation  2  is  the  resulting  equation  that  is  graphed  as  the  best  fit  curve 
for  the  data  collected. 


1  -  k  (2) 

P  ~  1  +  e-(B0+BiO  + 

where: 

p  =  probability  of  obtaining  the  correct  orientation  of  the  Landolt  C 
k  —  baseline  probability  (1  chance  in  4  of  getting  the  correct  orientation  by  guessing, 
i.e.,  0.25) 

£?!  =  slope  determined  by  the  linear  regression  of  the  logit  with  distance 
B0  =  intercept  determined  by  the  linear  regression  of  the  logit  with  distance 
d  =  distance  from  sensor  to  Landolt  C  target. 
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The  next  step  in  determining  a  resolution  value  for  each  sensor  is  to  select  a  criterion  point.  It  was 
decided  to  select  a  distance  value  that  corresponded  to  a  point  on  the  logistic  fit  curve  that  was  half 
way  between  chance  (0.25)  and  100%  correct  (1.00).  This  is  somewhat  similar  to  a  50%  probability 
threshold  curve  where  the  software  is  correct  (after  subtracting  the  25%  for  chance)  about  half  of 
the  time.  The  probability  value  that  corresponds  to  this  resolution  distance  is  half  way  between  0.25 
and  1.00  or  (1.00  +  0.25)/2  =  0.625.  Referring  back  to  Equation  1  we  see  that  when  we  substitute 
the  values  of  k  =  0.25  and  p  =  0.625  into  the  equation  we  get, 


fV~k\  ( 0.62b -0.2b\  /0.3/b\ 

Logit  =  ln  (_)  =  in  (  t_06z5  )  =  in  (o^)  =  ln(l)  =  0. 


'0.625  —  0.25 


0.375n 


(3) 


This  means  the  resolution  distance  is  that  distance  for  which  the  linear  best-fit  curve  expression  at 
the  right  side  of  Equation  1  equals  zero.  In  equation  form, 


ln  (H)  =  0  =  Bi  *  dR  +  Bo- 

which  can  easily  be  solved  for  dR,  the  resolution  distance  (Equation  5  below). 


where: 

dR  =  resolution  distance 
B0  =  intercept  of  the  linear  regression  equation 
=  slope  of  the  linear  regression  equation. 


(4) 


(5) 


The  resolution  distance  for  the  sensor  is  the  distance  at  which  the  Landolt  C  can  be  correctly 
interpreted  at  a  rate  half  way  between  chance  (25%)  and  certainty  (100%).  The  measured  sensor 
resolution  is  then  the  angular  subtense  of  the  Landolt  C  gap  size  (12.5mm)  at  this  distance,  which  is 
given  by  Equation  6. 

Measured  Sensor  Resolution  =  Rm  —  (12.5 /dR)  in  milliradians  (6) 

where: 

Rm  =  measured  resolution  in  milliradians 
dR  =  resolution  distance  in  meters. 


One  problem  with  using  Equation  1  for  curve  fitting  is  that  the  logit  is  undefined  for  any  data  point 
that  is  equal  to  or  below  0.25.  Luckily,  for  our  data  set  (Tables  3  and  5),  all  fraction  correct  values 
were  in  excess  of  the  0.25  chance  level.  If  this  had  not  been  the  case  we  would  have  had  to 
substitute  0.25  for  any  percent  correct  data  point  that  was  below  0.25.  The  logit  is  also  not  defined 
for  a  fraction  correct  value  of  1.00  so  a  value  of  0.999  was  entered. 


3.4  Sensor  noise  issue 

Of  the  four  sensors  selected,  it  was  apparent  that  the  NIR  sensor,  basically  the  same  spectral 
sensitivity  as  night  vision  goggles,  had  the  most  noise  at  the  illuminance  levels  used.  The  human 
visual  system  can  “average  out”  this  noise  to  some  degree.  Therefore,  it  was  decided  to  collect  and 
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average  10  consecutive  video  frames  from  the  NIR  sensor  to  simulate,  to  some  extent,  the  human’s 
visual  capability  for  each  image  that  was  used  for  assessing  the  NIR  sensor  resolution. 


3.5  Synthetic  sensor  (fused  images)  resolution 

Once  all  of  the  images  from  all  sensors  and  all  distances  were  collected,  it  was  possible  to  register 
and  fuse  images  collected  from  different  sensors  for  the  same  distances.  Since  the  fields  of  view  of 
the  sensors  varied  somewhat,  images  did  have  to  be  resized  in  order  to  register  and  fuse  the  images. 
Two  “synthetic”  sensors  were  evaluated  in  this  fashion:  the  combination  of  the  VIS  and  NIR 
sensors  and  the  combination  of  the  SWIR  and  LWIR  sensors.  It  should  be  remembered  that  for  this 
project,  the  Landolt  C  target  always  had  the  same  contrast  parity;  that  is,  it  was  always  a  dark  C  on 
a  light  background  for  all  sensors  making  it  somewhat  easier  to  fuse  the  two  images. 


4.0  RESULTS 


4.1  Basic  sensor  results 

Table  3  is  a  summary  of  the  fraction  correct  responses  from  the  ALCOR  software  for  the  four 
sensors  at  the  various  distances  used.  As  can  be  seen  from  this  table,  the  fraction  correct  is  very 
high  for  the  shorter  distances  for  each  sensor  and  then  drops  off  to  close  to  chance  only  (25% 
correct)  at  the  farthest  distances.  Figure  7  shows  graphs  of  the  data  shown  in  Table  3  along  with  the 
best-fit  logit  regression  curves.  Table  4  is  a  summary  of  the  calculated  resolution  for  each  sensor 
(based  on  number  of  horizontal  elements  and  the  horizontal  field  of  view)  compared  with  the  sensor 
logit  curve  fit  value  at  62.5  percent  probability  of  correct  response.  The  last  column  of  Table  4  is 
the  ratio  between  the  resolution  measured  using  the  logit  fit  curve  and  the  calculated  resolution. 


Table  3.  Fraction  correct  Landolt  C  orientation  determination  as  a  function  of  distance  for  each  sensor. 

The  column  labeled  “C-gap  Size”  is  the  angular  subtense  of  the  Landolt  C  at  the  distances  shown; 
_ _ these  can  be  compared  to  the  sensor-calculated  resolution  _ _ 


Sensor-calculated 

Resolution 

1 .96  mrad 

1.67  mrad 

C-gap  Size 
(mrad) 

Distance  (meters) 

LWIR 

SWIR 

6 

0.99 

0.98 

2.08 

7.5 

0.92 

0.99 

1.67 

9 

0.77 

0.99 

1.39 

11 

0.67 

0.98 

1.14 

13 

0.56 

0.67 

0.96 

16 

0.29 

0.63 

0.78 

19 

0.26 

0.66 

Sensor-calculated 

Resolution 

0.71  mrad 

0.75  mrad 

C-gap  Size 
(mrad) 

Distance  (meters) 

NIR 

VIS 

10 

1.00 

0.97 

1.25 

13 

0.92 

0.98 

0.96 

16 

0.87 

0.87 

0.78 

20 

0.58 

0.79 

0.63 

24 

0.39 

0.63 

0.52 

28 

0.36 

0.40 

0.45 

34 

0.26 

0.30 

0.37 
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LWIR  Sensor 


SWIR  Sensor 


Distance  (meters) 


VISIBLE  Sensor 


Figure  7.  Graphs  of  the  fraction  correct  responses  for  each  of  the 
four  sensors  as  a  function  of  distance  with  logistic  regression  curve 


Table  4.  Comparison  of  the  calculated  and  measured  resolutions  (milliradians  or  mrad)  and  the  ratio  between  the  two 


Sensor 

Calculated  Resolution 

Measured  Resolution 

Ratio 

(mrad) 

(mrad) 

SWIR 

1.67 

0.83 

0.50 

LWIR 

1.96 

1.08 

0.55 

VIS 

0.75 

0.53 

0.71 

NIR 

0.71 

0.58 

0.81 

4.2  Fused  “sensor”  results 

Table  5  is  a  summary  of  the  fraction  correct  Landolt  C  orientation  detennination  as  a  function  of 
distance  for  two  sensor  combinations:  the  SWIR-LWIR  and  the  VIS-NIR.  These  were  the  only  two 
combinations  that  could  be  tested  because  of  the  aforementioned  issue  with  images  being  collected 
at  different  distances  for  the  “low  resolution”  and  “high  resolution”  sensors.  The  images  from  the 
pairs  of  sensors  were  fused  in  two  different  ways:  using  a  simple  averaging  algorithm  and  using  a 
LaPlacian  algorithm.  Figure  8  shows  graphs  of  the  SWIR-LWIR  fused  data  of  Table  5. 
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Table  5.  Summary  of  percent  correct  as  a  function  of  distance  for  the  fused  sensor  image  combinations  shown 


SWIR-LWIR  Fused 

Distance  (m) 

LaPlacian 

Average 

6 

0.73 

0.8 

7.5 

0.85 

0.95 

9 

0.89 

0.98 

11 

0.69 

0.68 

13 

0.48 

0.49 

16 

0.32 

0.33 

VIS-NIR  Fused 

Distance  (m) 

LaPlacian 

Average 

10 

0.96 

0.93 

13 

0.99 

0.81 

16 

0.79 

0.79 

20 

0.55 

0.61 

24 

0.4 

0.49 

28 

0.26 

0.26 

34 

0.29 

0.35 

SWIR-LWIR  Fused  Avg 


SWIR-LWIR  Fused  LaPlacian 


Distance  (meters) 


Distance  (meters) 


Figure  8.  Graphs  of  the  SWIR-LWIR  fused  image  data  from  Table  5  with  Averaging  and  LaPlacian  fusion  algorithms 


It  is  apparent  from  the  graphs  of  Figure  8  that  the  fraction  correct  data  are  dropping  at  the  lowest 
two  distances,  which  is  not  intuitive.  Inspection  of  these  images  indicates  that  they  are  easily 
interpretable  to  human  vision.  This  low-end  roll-off  only  occurred  with  the  “low  resolution”  sensor 
fusion  and  is  slightly  evident  when  looking  at  the  SWIR  sensor  curve  as  well.  We  are  not  sure  why 
this  effect  occurred  but  it  is  obviously  a  quirk  of  the  ALCOR  software  that  needs  to  be  addressed  in 
the  future.  In  order  to  provide  a  reasonable  assessment  of  the  measured  resolution,  it  was  decided  to 
apply  the  logistic  regression  curve  fit  with  the  two  lowest  distances  deleted  for  the  SWIR-LWIR 
fusion  combinations.  This  only  has  a  small  impact  on  the  measured  resolution  (as  shown  in  the 
table  below)  but  it  provides  a  much  more  pleasing  curve  fit  as  indicated  in  the  graphs  of  Figure  9. 
These  two  modified  curve-fits  are  designated  SWIR-LWIR  fused  Average  B  and  SWIR-LWIR 
fused  LaPlacian  B  to  differentiate  them  from  the  curve  fits  that  use  all  of  the  data  points. 
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SWIR-LWIR  Fused  Avg  B 


SWIR-LWIR  Fused  LaPlacian  B 


o 

U 


Distance  (meters) 


VIS-NIR  Average  Fused 


Distance  (meters) 


o 

U 


Distance  (meters) 


VIS-NIR  LaPlacian  Fused 


Distance  (meters) 


Figure  9.  Fused  image  curves  using  data  from  Table  5 


Table  6.  Summary  of  measured  resolutions  for  the  fused  sensor  combinations; 
the  smaller  the  number  the  better  the  resolution 


Fused  Sensor  Combinations 

Resolution 

(mrad) 

SWIR 

0.83 

LWIR 

1.08 

SWIR-LWIR  LaPlacian 

1.15 

SWIR-LWIR  Average 

1.03 

SWIR-LWIR  LaPlacian  B 

1.06 

SWIR-LWIR  Average  B 

0.99 

Fused  Sensor 
Combinations 

Resolution 

(mrad) 

VIS 

0.53 

NIR 

0.58 

VIS-NIR  LaPlacian 

0.61 

VIS-NIR  Average 

0.66 

5.0  ANALYSIS 

5.1  Comparison  of  calculated  resolution  and  measured  resolution 

Previously  it  was  noted  that  the  sensor  element  count  for  the  VIS  and  NIR  sensors  exceeded  the 
frame-grabber  element  count,  which  would  impact  the  value  of  the  calculated  resolution  (field  of 
view/sensor  element  count).  One  way  to,  at  least  partially,  correct  for  this  is  to  substitute  the  frame- 
grabber  element  count  for  the  sensor  element  count  to  obtain  a  modified  calculated  resolution. 

Table  7  below  shows  these  modified  calculated  resolutions  for  the  VIS  and  NIR  sensors,  which  are 
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designated  VIS  FG  and  NIR  FG  with  the  “FG”  signifying  the  frame-grabber  element  count  was 
used  in  the  calculation. 


Table  7.  Same  as  Table  4  except  with  modified  calculated  resolutions  for  the 
VIS  and  NIR  sensors  to  account  for  the  limiting  effects  of  the  frame-grabber  (FG)  element  count 


Sensor 

Calculated 
Resolution  (mrad) 

Measured 
Resolution  (mrad) 

Ratio 

SWIR 

1.67 

0.83 

0.50 

LWIR 

1.96 

1.08 

0.55 

VIS  FG 

0.90 

0.53 

0.59 

NIR  FG 

0.85 

0.58 

0.68 

Using  the  calculated  resolutions  of  Table  7,  it  is  possible  to  do  a  linear  regression  to  determine  how 
well  the  calculated  resolution  can  predict  the  measured  resolution  obtained  using  the  ALCOR 
software  and  logit  regression.  Figure  10  is  a  graph  of  the  data  of  Table  7  with  the  linear  best-fit  line 
shown.  The  correlation  between  the  calculated  resolution  (with  the  frame-grabber  limitation 
modification)  and  the  measured  resolution  was  0.97.  This  indicates  the  calculated  resolution  does  a 
fairly  good  job  of  predicting  the  measured  resolution,  although  the  values  are  not  close  (the 
measured  resolution  is  about  50%  to  68%  of  the  calculated  resolution  as  indicated  by  the  “Ratio” 
column  of  Table  7). 


Measured  Resolution  vs.  Calculated  Resolution 


Figure  10.  Measured  resolution  versus  calculated  resolution  using  the  data  from  Table  7;  correlation  is  0.97 


Based  on  this  analysis,  it  appears  that  the  measured  resolution  is  reasonably  consistent  with  the 
calculated  resolution  for  the  sensors  and  the  range  of  resolutions  involved. 

5.2  Measured  resolution  of  “synthetic”  sensors 

The  next  question  is:  What  do  the  measured  resolutions  of  the  “synthetic”  sensors  (the  fused 
combinations  of  sensors)  look  like  compared  to  the  measured  resolutions  of  the  component  sensors? 
The  data  for  this  comparison  are  in  Table  6,  which  is  discussed  in  the  next  section. 
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6.0  DISCUSSION/CONCLUSIONS 


6.1  Comparison  of  calculated  resolution  versus  measured  resolution 

Although  it  was  not  a  primary  purpose  of  this  study,  the  relationship  between  the  sensor  resolution 
measured  using  the  ALCOR  software  approach  and  the  sensor  resolution  calculated  from  the  sensor 
pixel  count  and  field  of  view  is  of  interest.  The  measured  resolution  includes  any  effects  that  may 
be  present  that  could  affect  the  image  quality  of  the  sensor  such  as  the  lens  modulation  transfer 
function  (MTF),  sensor  element  noise,  and  optical  defocus  whereas  the  calculated  resolution  is  not 
sensitive  to  these  (and  other)  potential  defects.  However,  assuming  these  potential  defects  do  not 
have  a  substantial  impact  on  the  actual  resolution  of  the  sensors  we  selected  for  our  study  we  would 
expect  there  to  be  a  reasonable  correlational  relationship  between  the  two  resolutions.  Table  7  and 
Figure  10  demonstrate  there  is  a  fairly  strong  relationship  between  the  two  (R-squared  value  of 
0.95),  which  is  encouraging.  Ideally,  one  would  like  to  see  a  correlation  of  1.00  and  an  intercept  of 
0  for  the  linear  regression  line  between  the  two  resolutions.  We  obtained  a  correlation  of  0.97 
(close  to  1.00)  but  an  intercept  of  0.15  mrad  (larger  than  desired),  indicating  that  there  were 
probably  some  non-geometric  (pixel  spacing  and  lens  focal  length)  parameters  affecting  the  image 
quality.  From  Table  7  one  can  see  that  the  NIR  sensor  had  a  better  calculated  resolution  (based  on 
geometry)  than  the  VIS  sensor  but  the  VIS  sensor  had  a  better  measured  resolution  (ALCOR)  than 
the  NIR  sensor.  Based  on  visual  inspection  of  the  images,  this  is  almost  certainly  due  to  the  noise  in 
the  NIR  sensor  images  even  though  they  were  averaged  across  10  frames. 

6.2  Measured  resolution  of  “synthetic”  sensors 

Table  6,  showing  the  results  of  the  measured  resolution  (ALCOR)  for  the  different  fused  sensor 
combinations  and  fusion  algorithms,  is  the  basis  for  this  discussion.  As  can  be  seen  in  Table  6  both 
of  the  fusion  algorithms  (Average  and  LaPlacian)  for  the  high-resolution  sensor  combinations  (VIS 
and  NIR)  had  poorer  measured  resolutions  (higher  numbers)  than  either  of  the  component  sensor 
measured  resolutions.  This  is  most  likely  due  to  the  problem  of  having  to  register  the  images  from 
the  two  sensors  prior  to  fusion.  The  fused  resolutions  of  these  two  sensors  ranged  from  6%  to  25% 
worse  than  the  component  sensor  resolutions  depending  on  which  fusion  algorithm  and  which  base 
sensor  one  is  comparing  to.  Based  on  this  result,  one  would  expect  the  fused  images  of  this  sensor 
combination  to  have  a  lower  resolution  than  either  component  sensor  by  itself  but  this  does  not 
necessarily  mean  that  human  visual  performance  for  an  actual,  field-relevant  task  (such  as  target 
detection  or  target  recognition)  would  be  worse  with  the  fused  images.  Human  visual  performance 
depends  not  only  on  the  resolution  of  the  fused  images  but  also  on  the  scene-specific  image  content. 
The  infonnation  content  of  the  fused  images  depends  largely  on  the  scene  characteristics  and  it  may 
be  that  this  visual  infonnation  content  difference  between  the  two  component  sensors  could  be  more 
significant  than  the  loss  of  resolution  due  to  image  registration/fusion.  This  is  an  area  worthy  of 
further  study. 

The  results  for  the  SWIR-LWIR  “synthetic”  sensor  were  somewhat  different.  For  this  case,  the 
fused  sensor  resolution  ranged  from  an  improvement  in  resolution  of  about  8%  to  a  decrement  of 
about  28%  depending  on  which  baseline  sensor  is  being  compared  to  which  fusion  algorithm.  The 
major  difference  between  this  sensor  combination  and  the  “high  resolution”  sensors  combination  is 
the  difference  in  measured  resolution  between  the  two  sensors  that  are  fused.  The  difference  in 
resolution  between  the  SWIR  and  the  LWIR  sensors  was  about  27%  whereas  the  difference  in 
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resolution  between  the  VIS  and  NIR  sensors  was  only  about  9%.  This  might  help  explain  the 
“improvement”  effect  found  with  respect  to  the  LWIR  sensor  compared  to  the  SWIR-LWIR  fusion. 
Namely,  that  the  SWIR  sensor  resolution  (higher  resolution)  had  enough  of  an  impact  on  the  SWIR- 
LWIR  fusion  algorithms  that  it  “over-powered”  resolution  losses  from  the  registration  process  when 
compared  to  the  LWIR  sensor  resolution. 

For  the  SWIR-LWIR  “synthetic”  sensor,  the  Average  fusion  algorithm  was  about  6%  better  than  the 
LaPlacian  fusion  algorithm  but  for  the  VIS-NIR  “synthetic”  sensor,  the  LaPlacian  was  about  8% 
better  than  the  Average.  Given  that  we  do  not  know  the  level  of  repeatability  of  the  ALCOR 
software  resolution  measurement  procedure  and  the  small  percentages  noted  here,  there  is  no  clearly 
apparent  advantage  to  either  the  Averaging  algorithm  or  the  LaPlacian  algorithm. 

Overall,  it  appears  that  the  resolution  (as  measured  by  the  ALCOR  software  approach)  of  the 
“synthetic”  sensors  is  generally  worse  (larger  number)  when  compared  to  the  resolution  of  the 
component  sensors,  probably  due  to  the  registration  process. 

6.3  Future  research  -  comparison  of  ALCOR  resolution  with  human  visual  resolution 
assessment 

The  work  described  in  this  paper  is  the  first  half  of  a  two-part  program  to  compare  sensor  resolution 
assessment  accomplished  using  a  novel  software  algorithm  approach  (ALCOR)  with  sensor 
resolution  assessment  accomplished  using  a  panel  of  human  observers.  The  research  protocol  and 
review  by  the  human  use  review  committee  to  conduct  the  human  panel  research  has  been 
completed  and,  by  the  time  this  paper  is  published,  should  be  in  process.  All  of  the  measured  sensor 
resolutions  addressed  in  this  paper  will  be  measured  in  a  similar  fashion  (logistic  regression  curve 
fit  value  at  62.5%  probability)  but  with  a  panel  of  human  observers  viewing  the  images  to  determine 
the  orientation  of  the  Landolt  C  to  obtain  the  fraction  correct  data.  These  data  will  then  be 
compared  to  see  if  the  ALCOR  software  approach  provides  similar  results  to  the  human  panel 
approach. 

Assuming  the  results  of  the  future  research  described  immediately  above  is  successful,  then  another 
interesting  study  would  be  to  include  the  resolution  target  (the  Landolt  C  in  our  case)  within  sensor 
images  that  contain  operationally  relevant  targets.  These  images  could  then  be  used  in  simultaneous 
efforts  to  determine  their  resolution  using  the  ALCOR  software  methodology  and  human  visual 
performance  (probably  target  recognition)  using  the  operationally  relevant  targets  within  the  same 
images.  This  could  be  done  for  both  the  individual  sensors  and  the  “synthetic”  sensors  to  see  if 
resolution  values  can  predict  human  visual  performance  levels. 

Another  area  for  future  research  is  to  conduct  the  same  ALCOR  software-based  resolution 
assessment  procedures  on  sensors  that  have  highly  mismatched  resolutions  due  to  significant 
differences  in  sensor  element  counts. 

For  the  current  effort,  we  used  a  5X5  matrix  of  resolution  target  “shifts”  in  order  to  average  out 
potential  sampling  issues  with  respect  to  the  relative  position  of  individual  sensor  elements  and  the 
Landolt  C  gap.  It  would  speed  up  the  process  if  we  could  get  the  same  results  using  a  3X3  matrix  of 
image  shifts  to  cut  down  the  number  of  images  at  each  distance  from  100  (5X5  shifts  by  4 
orientations)  to  36  (3X3  shifts  by  4  orientations). 
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The  ultimate  goal  of  this  line  of  research  is  to  devise  an  objective  technique  to  reliably  predict 
fusion  algorithm  effectiveness  without  having  to  conduct  human-based  visual  performance  studies. 


6.4  Conclusions 

The  research  conducted  so  far  has  resulted  in  a  software  algorithm  (ALCOR)  that  produces  an 
objective  measure  of  sensor  resolution  including  “synthetic”  sensors  that  are  produced  by  fusing  the 
images  from  different  sensors  that  have  different  spectral  band  sensitivities. 

The  correlation  between  simple  geometrically  based  sensor  resolution  calculations  and  the  sensor 
resolutions  measured  using  the  ALCOR  software  algorithm  and  logit  regression  was  quite  good 
(R~  =  0.95).  This  helps  validate  the  basic  approach  of  the  ALCOR  software.  However,  there  is  no 
geometrically  based  sensor  resolution  for  “synthetic”  sensors  since  component  sensors  used  to 
produce  the  fused  images  may  have  different  basic  resolution  geometries  (sensor  element  spacing 
and  lens  focal  length).  The  ALCOR  approach  potentially  provides  a  means  of  detennining  the 
resolution  of  a  non-existent,  “synthetic”  sensor.  Future  work  will  have  to  address  the  repeatability 
of  the  ALCOR  approach. 

Based  on  our  limited  sample  size  (four  sensors  and  four  “synthetic,”  fused  sensors),  it  appears  that, 
in  general,  synthetic  sensors  have  poorer  resolutions  than  their  component  sensor  resolutions  which 
is  probably  due  to  the  registration  process.  This  is  another  area  for  future  work. 
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